Skip to content

h2: a loop is taken as stuck after half a WriteTimeout and at least 2 s (the wedge fallback cut healthy downloads on a loaded host) - #1202

Merged
FumingPower3925 merged 4 commits into
mainfrom
flakes9-loopstuck-floor
Nov 1, 2026
Merged

FumingPower3925 merged 4 commits into
mainfrom
flakes9-loopstuck-floor

Conversation

@FumingPower3925

@FumingPower3925 FumingPower3925 commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Refs #1170

What failed

Coverage root shard c of #1193 (run 41246492268, attempt 1; the rerun passed): TestH2GoTransportManyParallelDownloadsAreNotCut1170/adaptive, 7 of the 8 parallel 8 MiB downloads reset with INTERNAL_ERROR at 4 MiB (server log: resets by reason loop-blocked:4 rate:3, 1 episode of a stuck loop). It had failed three times on the l19s3 branch before.

Cause

The wedge fallback of #1180 (LoopWatch) cuts every pool-handler wait that has lasted a WriteTimeout while any event loop has been in one batch for max(WriteTimeout/2, 250 ms), judged by a single sample of the batch's wall-clock age at the waiter's own 1 s timer. A healthy loop of an instrumented or overloaded host has batches of 0.4 to 1.9 s, almost all of it one h2 connection's write-queue drain (drainH2Writes; the 8 downloads share one connection). So a loop that is slow, not blocked, switched the rate rule off and the 8 waiters, whose timers fire together, were cut together.

Reproduced in docker arm64 (--cpus 2, GOMAXPROCS=2, -race -covermode=atomic -coverpkg=..., the CI shard's flags): 0/48 subtests with no extra load, 0/48 with one busy loop, 7/72 with two busy loops (every one loop-blocked first; stuck_for 0.64 to 1.8 s against the 0.5 s threshold; the CI signature loop-blocked then rate reproduced). Without -race or coverage the longest batch under the same load was 324 ms.

Fix

  • minLoopStuck 250 ms -> 2 s: a loop is taken as stuck after max(WriteTimeout/2, 2 s). Unchanged for a WriteTimeout of 4 s or more (the 60 s default keeps its 30 s). No hot-path cost, portable. The price is the bound of a real wedge at a small WriteTimeout: max(WriteTimeout/2, 2 s) + WriteTimeout after the loop blocks (3 s at WriteTimeout 1 s, was 1.5 WriteTimeouts = 1.5 s; measured by the wedge cell 1.3 s before, 2.3 s expected after).
  • A loop or worker that ends without returning (runtime.Goexit in a sync handler, epoll, io_uring: runtime.Goexit in a synchronous handler ends the loop or worker goroutine it runs on, and every connection that loop owns goes unserved #848) now clears its LoopWatch stamp on the way out. It was left set, and every waiting pool handler of that engine was cut as loop-blocked for the rest of the process.
  • The stamp is cleared first in the exit path (before the exit report, whose Logger can block).
  • Docs: Config.WriteTimeout, the OutboundBudget comment, H2Config.LoopWatch.
  • Cells: the many-downloads cell runs the two engines one at a time (they starved each other's loops) and asserts that no wait was reset as loop-blocked before it asserts the downloads; the wedge cell asserts its bound as the formula; the threshold pair blocks the loop for 2.5 s (over) and 0.1 s (under).

Considered and rejected (design review): a thread-CPU/run-queue verdict (schedstat) is unsound under Go (a runnable loop goroutine without a P has a sleeping thread, the same signature as the wedge: two healthy batches of 801 and 835 ms read ~0), and a progress counter has healthy CPU-busy batches of up to 4.1 s without a bump (one frame's growslice/GC assist).

Failing first

Commit 1 (tests only) is red on the parent code: TestLoopStuckThresholdHasAFloor1170, TestHealthyLongBatchIsNotAWedge1170 (0.8, 1.5, 1.9 s at WriteTimeout 1 s), TestWaiterLoopBlockedFallback1170/just_under, TestLoopWatch1170, and the Goexit stale-stamp assertions in the epoll and io_uring 848 tests. On mono.Manual, so the verdicts are exact. Green before and after: TestBlockedLoopIsCutWithinTheBound1170 (the wedge is still cut, within [threshold, threshold + WriteTimeout], at WriteTimeout 1 s, 5 s and 60 s) and the 60 s edge.

Verification

Docker arm64 (kernel 7.0.12-linuxkit), --cpus 2, GOMAXPROCS=2, --security-opt seccomp=unconfined --ulimit memlock=-1, golang:1.27.3, one container at a time; the test binary is built as the coverage job builds it (go test -c -race -covermode=atomic -coverpkg=github.com/goceleris/celeris ./test/server) and run as -test.run '^TestH2GoTransportManyParallelDownloadsAreNotCut1170$' -test.count=N; "busy loops" are while :; do :; done in the same container (noisy neighbours). One iteration = the epoll and adaptive subtests (a failure is counted per subtest). Logs and the driver scripts: the FLAKES9 evidence directory.

The cell, main against the fix (the cell unchanged, both subtests in parallel, unless stated):

run code busy loops iterations failed subtests reasons
earlier session, instrumented main f067838 0 24 0/48
earlier session, instrumented main f067838 1 24 0/48
earlier session, instrumented main f067838 2 24 + 12 7/72 loop-blocked first, stuck_for 0.64 to 1.8 s; CI shape (loop-blocked then rate) seen once
this session main 70444cb 2 12 + 24 0/72
this session main 70444cb 3 12 3/24 (epoll) loop-blocked:2, loop-blocked:5 rate:1, loop-blocked:2; 1 episode each
this session fix (cell unchanged) 3 12 0/24 0 loop-blocked resets, 0 episodes in all 24
this session fix (cell unchanged) 2 24 0/48 0 loop-blocked resets, 0 episodes; nothing regressed (main was also 0/72 at this load)
this session fix + serialised cell 2 24 0/48 0 loop-blocked resets, 0 episodes; nothing regressed (main was also 0/72 at this load)
this session fix + serialised cell 1 6 0/12 nothing regressed

The baseline did not reproduce at two busy loops this session (0/72 on main, against 7/72 in the earlier session, which was run with a build instrumented per batch; the host load differs from one session to the next). At three busy loops it did (3/24 on main, all loop-blocked), and the fix is 0/24 at the same load with the cell unchanged, so the fix is verified independently of the cell change. The deterministic evidence is the red/green unit tests below, not these rates. (3 failures in 24 against 0 in 24 is p about 0.1 by itself.) The Opus review ran an extra interleaved A/B at three busy loops with the no-loop-blocked assertion on both sides: main 3/32 failed (one with the exact CI signature loop-blocked:4 rate:3), this head 0/32; pooled with the 3/24 against 0/24 above: main 6/56 against 0/56, p = 0.014. The 2-busy-loop rows of the fix show only that nothing regressed; the evidence that the fix matters is the 3-busy-loop rows, the pooled A/B and the red/green unit tests.

Wedge and threshold cells (this head, 2 busy loops, 3 iterations of both tests):

  • TestH2PoolWaiterHoldingALockDoesNotWedgeTheServer1170: the locker is answered in 2.30 to 2.32 s on epoll, io_uring and adaptive, all 24 trials per engine (the assertion is the formula max(WT/2, 2 s) + WT + 1 s = 4 s; main measured 1.30 s). Wedge bound at WriteTimeout 1 s: 3 s.
  • TestH2PoolWaiterLoopBlockedThreshold1170: under (0.1 s blocks): 0 cut, 0 loop-blocked resets, epoll and adaptive; over (2.5 s blocks): 3 of 4 cut, loop-blocked and an episode told, both engines. The test takes 59 s (was 54.6 s in the CI log of iouring: the header deadline of a silent connection is a kernel timer armed at the accept; document the bound (Refs #1189) #1193).
  • Goexit stale stamp (TestSyncHandlerGoexitEndsOneLoopLoudly848, ...OneWorkerLoudly848, which check the dead loop's own slot; -race -count=20 in the container: 20/20 on both engines): red on the tests-only commit ("the loop that ended without returning left its LoopWatch stamp set"), green on the fix, both engines.
  • Unit tests (stepped clock): red on the parent code as listed above, -race -count=20 clean, clean with CELERIS_MONO_SKEW=3600s and -3600s (-tags celeris_monoskew); the whole stream package -race passes.
  • All 1170 cells of ./test/server once under -race -covermode=atomic in the container: pass, 494.7 s (the many-downloads cell is 67 s now, serialised, was 34 s: shard c has 660 s of timeout for about 360 s of work, so +34 s and +4 s stay inside the headroom).
  • golangci-lint clean on the changed packages (two errcheck hits in unchanged files of other packages come from a newer local lint version than CI's).

Release note

HTTP/2 pool-handler waits: the fallback that bounds a waiting handler by WriteTimeout while an event loop is blocked (#1170) now needs the loop to be in one batch for half a WriteTimeout and at least 2 s (was 250 ms). A loop that is merely slow (an overloaded or instrumented host) no longer switches the minimum-rate rule off; the waits of a handler that holds a lock a blocked loop waits for are still cut, at most max(WriteTimeout/2, 2 s) + WriteTimeout after the loop blocks (1.5 WriteTimeouts for a WriteTimeout of 4 s or more, 3 s at WriteTimeout 1 s); the 60 s default is unchanged. A loop that ends abnormally (runtime.Goexit in a handler) no longer leaves the fallback switched on for the rest of the process (the waits of that engine were all cut as loop-blocked).

Follow-up (not in this PR)

drainH2Writes appends up to 8 x 4 MiB into one connection's writeBuf in one pass (a 32 to 64 MiB growslice, its copy under the race detector, the GC assist it pays, then flushWrites' compaction copy), which is where the long batches come from. A bounded pass (stop pulling from the shard queue past a few MiB unsent, resume on EPOLLOUT / the next pass) would shorten the batches and the per-connection memory by an order of magnitude, but touches write-path accounting (pendingBytes/writeCap refusal, the drain clock) and io_uring's queue-to-SQE path. To be filed as a performance issue.

…mp; the many-downloads cell asserts the condition (red on main)

Refs #1170. Coverage root shard c, run 41246492268: TestH2GoTransportManyParallelDownloadsAreNotCut1170/adaptive
had 7 of 8 downloads reset (loop-blocked 4, rate 3). The loop-stuck fallback judges a loop stuck from the
wall-clock age of its current batch (max(WriteTimeout/2, 250 ms)); a healthy loop of an instrumented or
overloaded host has batches of 0.4 to 1.9 s (one h2 connection's write-queue drain), measured at 7 of 72
cells with -race, -covermode=atomic and two busy loops on two CPUs.

Unit tests, on mono.Manual (exact, no sleep):
- loopStuckFor has a 2 s floor (1 s -> 2 s, 5 s -> 2.5 s, 60 s -> 30 s unchanged);
- a batch of 0.8, 1.5 and 1.9 s at WriteTimeout 1 s is not a wedge, 2 s is; at 5 s and at the 60 s default
  the edge is 2.5 s and 30 s; a waiter that has not waited a WriteTimeout is never cut by it;
- a loop that blocks mid-wait is cut within [threshold, threshold + WriteTimeout] (the positive control,
  with the bound stated);
- a loop that ends without returning (runtime.Goexit, celeris#848) leaves its LoopWatch stamp set: every
  waiter of the engine would be cut as loop-blocked for ever (epoll and io_uring, in the 848 exit tests).

Cells: the many-downloads cell runs the engines one at a time and asserts that no wait was reset as
loop-blocked before it asserts the downloads; the wedge cell asserts the bound as a formula; the threshold
pair blocks a loop for 2.5 s (over) and 0.1 s (under).

Red on this commit's parent code: the floor, the healthy-batch cases, the stale stamp. Green: the bound,
the 60 s edge, the cells.
… s; a loop that dies leaves no stamp

Refs #1170. The wedge fallback (LoopWatch) cuts every pool-handler wait that has lasted a WriteTimeout
while any event loop has been in one batch for max(WriteTimeout/2, 250 ms). A healthy loop of an
instrumented or overloaded host has batches of 0.4 to 1.9 s (one h2 connection's write-queue drain), so the
cell of eight parallel downloads at WriteTimeout 1 s lost streams as loop-blocked (Coverage root shard c,
run 41246492268). The floor is now 2 s (minLoopStuck): unchanged for a WriteTimeout of 4 s or more (the
60 s default keeps its 30 s), and a wedge is cut at most max(WriteTimeout/2, 2 s) + WriteTimeout after the
loop blocks (3 s at WriteTimeout 1 s, was 1.5 WriteTimeouts). No hot-path cost.

A loop or worker that ends without returning (runtime.Goexit in a sync handler, celeris#848) now clears
its LoopWatch stamp on the way out: it was left set, and every waiting handler of the engine was cut as
loop-blocked from then on.

Docs: Config.WriteTimeout, the OutboundBudget comment and H2Config.LoopWatch state the threshold and bound.
@FumingPower3925 FumingPower3925 added this to the v1.6.0 milestone Nov 1, 2026
@coderabbitai

coderabbitai Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@codecov

codecov Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 75.00000% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
internal/protocol/h2/stream/write_wait.go 50.00% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

… exit clears the stamp first

Refs #1170. Review r1 of #1202:
- MINOR-1: BlockedFor(now+1h, 1) is true when ANY slot of the engine is stamped, so a live loop caught in
  a batch at that instant read as the dead loop's stamp (a flake shape on a starved runner). The tests find
  the loop or worker the /boom ran on (boomID848) and check its own slot (LoopSlot.Stamped).
- NIT-1: the Leave ran after reportLoopExit/reportWorkerExit, whose Logger can block while the stamp ages
  and the engine's waiters are cut as loop-blocked. It is now the first statement of the deferred function.
@FumingPower3925
FumingPower3925 marked this pull request as ready for review November 1, 2026 14:59
@FumingPower3925
FumingPower3925 merged commit 0db042d into main Nov 1, 2026
33 of 34 checks passed
@FumingPower3925
FumingPower3925 deleted the flakes9-loopstuck-floor branch November 1, 2026 15:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant